Update the documentation to use parquet output#2607
Update the documentation to use parquet output#2607erikvansebille merged 83 commits intoParcels-code:mainfrom
Conversation
Covered by test_write_dtypes_pfile
for more information, see https://pre-commit.ci
Remove temporary test_cftime.py file
This function is now independent of the time_interval as time is now stored as float
Remove nested key - save on root instead
VeckoTheGecko
left a comment
VeckoTheGecko
left a comment
There was a problem hiding this comment.
As part of review I've looked both at the code, as well as visually compared plots before and after.
I've gone through and pushed some edits which were quite straight-forward:
- 1b35bf9 Fixing a notebook
- d977c88e7
Now the docs builds are passing
Other than that, I have some small comments - nothing major.
Given we're now using Polars in the docs, the tests, and in the read_particlefile function - I think its easiest just for us to add it as a core dependency to Parcels. We could make it an optional dependency, but we don't really have the tooling for that in Parcels (and I don't think its worth adding the tooling in this case).
If we add as a core dependency:
- Update
pyproject.tomlandpixi.toml(run-dependenciesto= ">=1.31.0"andfeature.minimum.dependenciesto= "1.31.*") - Update
recipe.yaml
I'm happy to make those updates.
| assert isinstance(df["time"][0], (cftime.datetime, datetime)), ( | ||
| "CF-time values in Parquet did not get properly decoded. Are the attributes correct?" | ||
| ) |
There was a problem hiding this comment.
This assert should be updated pending the discussion from the other comment on the read_particlefile function.
There was a problem hiding this comment.
I've gone through this - really nice update! I think its quite clear
Now that each trajectory is plotted by itself, colour range has to be set for all
Co-authored-by: Nick Hodgskin <[email protected]>
Do you agree with this @erikvansebille ? |
|
Yes sorry, missed that comment about the polars dependency, and merged too fast. Can you make these updates in a new PR? |
|
All good, can do :) |
2 participants
Description
This PR updates all the documentation and tutorial notebooks to parse the parquet output introduced in #2600, as tracked in #2582. It also updated the
parcels.read_particlefile()to use polars, which scales better for large output filesChecklist
mainfor normal development,v3-supportfor v3 support)AI Disclosure